Visualizing Events: Simulating Meaning in Language

Background In this note, we present results from a new approach to modeling the semantics of natural language, Multimodal Semantic Simulations (MSS), being pursued by the first author and his students. This approach assumes both a richer formal model of events and their participants, as well as a modeling language for constructing 3D visualizations of objects and events denoted by natural language expressions. The Dynamic Event Model (DEM) encodes events as programs in a dynamic logic with an operational semantics, while the language VoxML, Visual Object Concept Modeling Language, is being used as the platform for multimodal semantic simulations in the context of human-computer communication. For over two decades, Pustejovsky has been investigating the nature of events in language: the Aktionsarten and event typing associated with lexemes and phrases; how they are mapped to syntax; and how nominal semantics interacts with type coercion behaviors involving events, as modeled in Generative Lexicon. Generative Lexicon has developed a fairly sophisticated model (both data structures and mechanisms) for how the function and use of an object can be encoded as part of its lexical semantics. In addition, he has worked to develop open standards for encoding the temporal and spatial information associated with events. The resulting languages, ISO-TimeML and ISOspace, are ISO standards for their areas, and ISO-TimeML has become the most widely adopted language within the field for representing temporal information in open domains.